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This study was designed to: (1) assess how evaluation 
recommendations have been addressed by decision makers and 
administrators; (2) examine the impact of evaluations on program 
development; (3) identify factors affecting the utilization of 
c^valuation findings and recommendations; and (4) provide 
recommendations to enhance utilization of information for future 
evaluation by the Office of Educational Accountability (OEA). A 
sample of six program evaluations, representing the full range of OEA 
evaluation activities, was selected from those conducted during 
1982-83 and 1983-84. Data were gathered by means of: (1) open-ended 
in-depth interviews with the evaluators and administrators involved 
in the six evaluations; (2) a telephone survey of school-level 
program staff of one of the six programs; and (3) a documentation 
review. The following recommendations were made: (1) institute 
procedures to follow up major evaluations after the final report; (2) 
involve administrators in a review of findings; (3) offer program 
implementers orientation and opportunity for input into the process; 
(4) increase the dissemination activities undertaken by OEA; (5) 
provide each school with a data summary; (6) establish a policy for 
preparing recommendations by OEA evaluators that incorporate 
effective criteria; and (7) incorporate into the OEA 1985-86 
follow-up study an analysis of how administrators address and use 
evaluation information. Appendices include a description of the six 
selected evaluations, a table showing iiq;>lemeBtation status of 
recommendations from six evaluation reports, and an overview of the 
use of evaluations in education. (JAZ) 
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EXECUTIVE SUMMARY 



Evaluations of DCPS programs are conducted by the Program Evaluation Depart- 
ment of the Office df Educational Accountability (OEA) to provide Information 
to decision makers that will help them determine If programs should be con- 
tinued* modified or eliminated. Evaluations also are expected to provide 
recommendations that serve ae guidelines for the future operation of evaluated 
programs. 

The study reported here was desxgned to: 1) assess how evaluation recommenda- 
tions have been addressed by decision makers and program implementers; 2) ex- 
amine the impact of evaluations on program development; 3) identify factors 
affecting the utilization of evaluation findings and recommendations; and 4) 
provide reconmiendatlons to enhance utilization of information for future OEA 
program evaluations. In this study, special attention was given to the role 
of program staff in the evaluation process, dissemination of final reports, 
and follow-up activities conducted by OEA. 

A sample of six program evaluations, representing the full range of OEA eval- 
uation activities, was selected from those conducted during the 1982-83 and 
1983-84 school years. Data were gathered by means of: 1) open-ended in-depth 
interviews with the evaluators and program administrators involved in the six 
evaluations; 2) a telephone survey of school-level program staff of one of the 
six programs; and 3) a documentation review. 

Findings 

All recommendations from the six evaluations comprising this study were re- 
ported as having been addressed, and 84% had been completely or partially 
implemented within one year following publication of the final evaluation 
report. The most important keys to implementation were the project manager *s 
commitment to the evaluation and his or her authority to implement recommenda- 
tions. 

The majority of implemented recommendations dealt with activities and proce- 
dures program managers could carry out under their own authority, and did not 
require major budgetary changes. Fully implemented recommendations were those 
which tended to identify general program needs, rather than specific ones, and 
mainly involved increasing or enhancing ongoing program activities, rather 
than starting new ones. 

Program managers held positive expectations about the eventual impact of the 
evaluations, and these expectations were largely confirmed. Several kinds of 
impact were found. Evaluations: 1) helped improve the definition, direction 
and scope of activities in the programs; 2) provided information on the effec- 
tiveness of existing procedures; 3) identified problems in record-keeping, 
leading to tightening of record management procedures; and 4) provided support 
for continued funding. 

A major factor contributing to evaluation utilization ws.q the participation of 
program managers in the evaluatiosj process. They, and otl>er program adminis- 
trators., participated in the formulation of evaluation objectives, in the 
planning of the evaluation, in the development of data-collection instruments, 
and in the gathering of data. This ensured that the evaluations addressed 
relevant prograumatic issues and ensured that findings had direct utility for 
program development and decision making. 
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While positive about their role In the evaluation process* program administra- 
tors suggested that they also be Invited to participate In preliminary reviews 
of findings and recotmnendations* with the evaluator» while the report was 
being prepared. Similarly » It was suggested that program Implement ers be 
given more extensive orientation at an early point In the process and more op- 
portunity CO provide Input Into the evaluation. 

Dissemination of evaluation information appeared to be limited in its scope 
and its Impact on recipients of reports. It was felt by program staff that 
more effective dissemination and other OEA follow-up activities were needed, 
and that these could be beneficial to both the evaluation process and imple- 
mentation practices. In addition to written materials from OEA, discussions 
and other oral presentations of evaluation Information were viewed as particu- 
larly effective possibilities. 

Recommendations 

1. Institute procedures to routinely follow-up all major evaluations after 
dissemination of the final report. 

2. Involve program administrators Ir a preliminary review of the findings 
and recommendations. 

3. Offer program Implementers more extensive and earlier orientation and op- 
portunity for input into the evaluation process. 

4. Increase the number and kinds of dissemination activities undertaken by 
OEA. 

5. Provide for each participating school a summary of the data collected at 
that school. 

6. Consider establishing a policy for preparing recommendations by OEA eval- 
uators that Incorporates criteria which have proven effective. 

7. Incorporate into the proposed OEA 1985-86 follow-up study an analysis of 
how different levels of administrative and program staff address evalua- 
tion recommendations and utilize evaluation information. 

Adoption of these recomomendations would result in an expansion of OEA program 
evaluation staff activities and Increased costs for carrying out evaluations. 
This would impact on the number of evaluations ttat OEA could undertake with 
Ics present resources. 
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INTRODUCTION 



Evaluations of educational programs are carried out by the Office of Educa- 
tional Accountability (OEA) at the request of Dade County Public School (DCPS) 
administrators, the School Board and In cov^pllance with federal and state re*- 
qulrements, Basic products of evaluations are findings* and the recommenda- 
tions based on those findings. The evaluation Information Is expected to 
serve decision makers who must decide If a program Is to be continued i modi- 
fied or eliminated, and to provide guidelines for future program implementa- 
tion. 

Reported here Is a follow-up study designed to assert the degree to which 
evaluation findings and recommendations have been addressed by decision makers 
and program implementers in the Dade County Public Schools. Factors affecting 
the timely utilization of program evaluations were also explored, along with 
the impact of the evaluation process on program development. Based on the re- 
sults of this study, recomnendations are presented to enhance the utilization 
of evaluations carried out by OEA. 

EVALUATION QUESTIONS 

This follow-up study had two major objectives: 1) to determine evaluation im- 
pact on programs by assessing the degree to which recommendations were ad- 
dressed and by identifying other outcomes the evaluation had on program devel- 
opment; and 2) to identify procedures that will enhance the usefulness of 
evaluation for program development and policy decisions. 

The following questions were considered regarding the recommendations and 
evaluation impact from the sample of selected program evaluations: 

1. To what extent were evaluation recommendations addressed by program 
managers? 

2. What factors influenced the utilization of program evaluation find- 
ings and recommendations? 

3. What kinds of impact did the evaluations have in addition to changes 
brought about by the implementation of evaluation recommendations? 

The following questions were considered regarding evaluation procedures and 
enhancement of evaluation effectiveness and utility: 

4o What kind of involvement did program administrators and staff have 
in the evaluation process? 

5. What evaluation procedures might be included to enhance evaluation 
utilization? 

6. Are current dissemination and follow-up procedures effective for 
ensuring that program staff understand findings and recemmendations? 

7. What dlsseminat5on and follow-up procedures might enhance implemen- 
tation of recommendations? 
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METHODOLOGY 



Sample of Program Evaluations 



Six program evaluations were randomly selected from those conducted during the 
1982-83 and 1983-84 school years. The final sample represents: 1) various 
funding sources (Federal, State and local); 2) different sources of 
evaluation requests (Federal/State requirements. Administrative Staff, and The 
School Board); 3) Internal (OEA) and external (contracted) evaluations; and 4) 
different time Intervals since the evaluation reports were completed and 
disseminated. 

Table 1 presents a summary of the characteristics of the six selected proj- 
ects. A description of each evaluation study Is presented In Appendix A. 
Three of the six evaluations were conducted during the 1982-83 school year and 
three during the 1983-84 school year. The amount of time from the completion 
of the evaluation report to the present follow-up study varied from eight 
months (College Assistance Program, CAP) to twenty-one months. (Beginning 
Teacher Program, BTP). 

Three of the six evaluations were conducted on new programs that were evalu- 
ated in subsequent years. The other three evaluations involved ongoing pro- 
grams; of those, two were evaluated in the following school year. 

Five of the six evaluations were designed to look at program effects on stu- 
dents, and in some instances or parents and staff. These were all evaluations 
of Bureau of Education programs. The sixth evaluation was of a nev program in 
the Bureau of Staff Development, and was designed to assess the effect of the 
program on developing the professional competence of beginning teachers (BTP). 

Five of the six evaluations were conducted by the Program Evaluation Depart- 
ments An external evaluation team carried out The Dropout Reduction and Pre- 
vention Program evaluation (Project Success). 

In summary, the six evaluation studies selected from those conducted In the 
school years 1982-^83 and 1983-84, represent the full range of evaluation ac- 
tivities undertaken by the Program Evaluation Department, 

Procedures 



Three kinds of procedures were used to conduct this follow-up study. They 
were^ 1) review and analysis of project evaluation related documentation; 2) 
open-ended in-depch interviews; and 3) telephone surveys. 

Docinnentation Review 

These procedures Included a review and analysis of documentation related to 
each of the six evaluations. The documents included: administrative staff 
responses to reports g evaluation reports from succeeding years, evaluation 
needs assessment » and other relevant documents as available. 



TABLE 1 



PROGRAM 



COLLEGE 

ASSISTANCE 

(CAP) 



ACADEMIC 

EXCELLpCE 

(AEP) 



CHARACTERISTICS OF THE EVALUATIONS IN THE FOLLOW-UP STUDY 



EVALUATION 
REQUESTED BY 



TARGET PROGRAM STATUS SCHOOL YEAR DATE REPORT 

POPEATION AT EVALUATION: EVALUATION PUBLISHED 
NEW/ONGOING CONDUCTED 



ENGLISH FOR 
SPEAKERS OF 
OTHER LANGUAGES: 
(ESOL) 

Oral Fluency of 
Linlted Eogllsh 
Proficient 12th 
Grade Students 



DCPS 

Administrative 
Staff 



DCPS 



Eleventh and 
Twelfth Grade 
Students, 
District-wide 



Ongoing 



Academically New 



Administrative Above Average 



Staff 



School Board 



Elementary 
Students 
Enrolled in AEP 
in 24 Schools 

LEP Twelfth 
Grade Students 
Enrolled in 
ESOL, 

District-wide 



Ongoing 



1983-84 



1983-84 



1983-84 



SUBSEQUENT 

PROGRAM 

EVALUATION(S) 



October, 1984 



September, 1984 1984-85 



August, 1984 



1984-85, 
Evaluation of 
ESOL Exit Cri- 
teria for 
Senior High 
Students 



DROPOUT 
PREVENTION 
(Project 
Success) 



COMPUTER 

EDUCATION 

(CEP) 



Federal/State 
Requirement 
(ECIA, Chapter 
11) 



Federal/State 
Requirement 
(ECIA, Chapter 
II) 



High Risk New 
Students in 
Grades 9-12 in 
Four Senior High 
Schools 

Elementary/ Ongoing 

Secondary 

Students 

in 132 Schools 



1982-83 



1982-83 



February, 1984 1983-84 
1984-85 



December, 1983 1983-84 
1984-85 



BEGINNING 

TEACHER 

(BTP) 



"ERIC to- 



State of Florida Elementary/ 
Secondary 
Beginning 
Teachers, 
District-wide 



New 



1982-83 



September, 1983 1983-84 
1984-85 
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Interviews 



In-depth open-ended Interviews were held with the evaluator and program man- 
ager of each of the evaluations included In the follow-up study. In addition » 
higher-level program administrators with supervisory responsibility over three 
projects (CAP, Success, BTP) were Interviewed. These Interviews were con- 
ducted because Implementation of some of the recommendations could not be 
carried out by the program manager without authorization and active support of 
these higher-level administrators. Although Implementation of some recommen- 
dations was dependent on decisions by the highest level decision makers 
(assistant /associate superintendents, the Superintendent of Schools, and the 
School Board), they were not Interviewed for this follow-up study. 

Also, for one evaluation. Project Success, the Interview process was extended 
to Include staff who had had major responsibility for program Implementation 
In 1982-83 and 1983-84 although they were not designated as program managers. 
These staff members had responsibility for coordinating activities as well as 
directly working with students. In addition the designated program manager 
changed from 1982-83 to 1983-84, and therefore both persons were Interviewed. 

These Interviews covered the following topics: development of evaluation 
objectives and evaluation plan; Involvement of program staff In the design of 
the evaluation plan; program administrators* expectations of, and actual Im- 
pact of the recommendations; dissemination and follow-up of evaluation 
reports; review of evaluation recommendations in terms of implementation 
status and identification of program persons responsible for their implementa- 
tions; dissemination and follow-up activities; and suggestions for modifica- 
tion of the evaluation and dissemination process. 

Interviews ranged in length from 45 minutes to three hours, with an average 
length of two hours. 

Telephone Survey 

A follow-up telephone survey was conducted of a randomly selected sample of 
teachers responsible for the implementation of evaluation recommendations in 
one project, the Academic Excellence Program. This project was selected 
because specific evaluation recommendations directly required actions by AEP 
principals and teachers for their implementation. The purpose of the survey 
was to assess the status of each recommendation that Involved staff partici- 
pation for implementation. Other purposes Included determining how staff per- 
ceived the evaluation process and their knowledge of the findings and recom- 
mendations included in the evaluation report. 

The survey was carried out in eight schools « Schools were selected from those 
that had the AEP program in 1983-84 and 1984-85 and had the same AEP teacher 
both years, or the AEP teacher for 1983-84 and the AEP teacher for 1984-85 
were both available for interviewing. The average length of the telephone 
interview was 15 minutes. All schools and teachers contacted participated in 
the sur^'ey. 
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FINDINGS 



Follow-up of Evaluation Recommendations 

This section examines two related issues: 1) to what extent the evaluation 
recommendations were addressed, and 2) what factors influenced the utilization 
of the recommendatione. 

The data presented here were gathered through interviews* a telephone survey 
and a documentation review. All principal evaluators were interviewed. The 
final sample of program administrators and staff represents persons with 
divect responsibility for carrying out the recommendations contained in the 
six evaluations. 

Characteristics of the RecOiiaaiendations 

The total number of recomr^endations in the six evaluations was 36; the number 
of recommendations per evaluation ranged from two to eight. The majority of 
them dealt with program operations and development issues that were the 
responsibility of the program manager; that is to say, implementation did not 
require active involvement from decision makers at higher levels in the 
administrative hierarchy. Recommendations requiring input from higher-level 
administrators were tho^e that Implied budgetary changes or interdepartmental 
collaboration for successful implementation. 

Recommendations varied in the extent to which they specified a set of actions 
to be taken, but tended to describe general kinds of actions rather than 
specific ones. An example of a general type of recommendation was: "Some 
type of credit or other reward needs to be considered for the teachers involv- 
ed in the project, especially team leaders." An example of a relatively spec- 
ific recommendation was: "Teachers who teach the after-school programs should 
receive equitable compensation for extra time required." 

Implementation Status 

The status of each recommendation was assessed using a coding system adapted 
from two studies designed to measure the utilization of evaluation and 
consultant recommendations in mental health programs (Anderson, Ciarlo and 
Brodie, 1981; Larsen and Werner, 1981). The categories used in the coding 
system were modified to reflect the specific context of educational 
programming in the Dade County Public Schools System. Table 2 presents the 
categories and their definitions. 
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TABLE 2 



Utilization Categories and Their Definitions 

1. Considered and rejected ^ Discussion of the recoxmnendation took place. 
The decision was made that action would not b3 taken on the 
recommendation, or that action was not possible at that time. 

2. Under cons iderat ion . Discussion of the recommendation took place, or 
action was pending further information. 

3. Partially implemented or implemented in modified form . Decision was taken 
to implement the recommendation. It was in the process of being 
implemented, was implemented only in some schools, or certain features of 
the recommendation were modified to adapt to program needs. 

4. Implemented as presented . This category includes the following: a new 
action, practice or policy; an intensification of a pre-existing action, 
practice or policv; or confirmation of an action taken prior to the 
presentation of evaluation recommendations. 



In Appendix B, a summary of the implementation :5tatus of each of the 36 
recommendations made by OEA evaluators for the six programs is presented. The 
basis for analysis was the implementation status of the 36 recommendations 
within a year following the publication of each of the six reports. For one 
program (Project Success), the implementation status is presented for the two 
years following the evaluation in 1982-83. These data illustrf^te how some 
evaluation recommendations require more than one school year for full or even 
partial implementation. However, in the analysis of the implementation status 
of the recommendations for Project Success, only the outcomes of the twelve 
months following the evaluation were included. 

As can be seen, all recommendations were addressed by program managers. The 
vast majority, 84%, were implemented in some form, and less than 10% were re- 
jected: 

Considered and rejected 8% 

Under consideration 8% 

Partially implemented 28% 

Implemented as presented 56% 

There are no established yardsticks to indicate how these percentages compare 
with utilization results elsewhere. However j the fact that all were addres- 
sed, and implementation of over 80% begun within a year, shows that OEA recom- 
mendations receive priority consideration among program managers. 
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of the three recommendations rejected, one was a possible misinterpretation by 
program administrators, one was not funded — although both the program manager 
and his director were in agreement and requested additional funding, and one 
was not possible because of physical space limitations. Actions on the other 
recommendations not Implemented at the time of this follow-up study are pend- 
ing the outcome of evaluation studies conducted during the 1984-85 school 
year. 

There were tea recommendations partially Implemented. Of these, four were Im- 
plemented In some schools but not others, and six were being Implemented but 
In a manner at variance with that suggested In the recommendations. In many 
cases it was a budgetary constraint that limited the scope of Implementation; 
In these cases, project managers were In agreement with the recommendations 
but were unable to provide the funds necessary to carry them out. 

Among the 20 recommendations categorized as being Implemented as presented, 
30Z Identified a new action that could be taken to Improve the programs. The 
majority, 60%, were calls for more of the actions already underway In the pro- 
grams, and 10% were confirmation of actions already taken by the program man- 
agers. 

In summary, all the evaluation recommendations had been addressed by program 
managers, and the majority had been completely or partially Implemented within 
a year following the publication of the evaluation report. This high degree 
of implementation reflects the fact that: 1) the program managers were in 
agreement with the majority of recommendations; 2) many of the recommendations 
required enhancing procedures already in place; and 3) the majority of recom- 
mendations provided general guidelines, leaving the specifics of implementa- 
tion up to the program staff. 

Impact of Evaluations 

In this study, the utility of the evaluations wa3 examined not only by looking 
at the implementation of recommendations, but also by trying to determi^.a 
other kinds of program impact. In order to accomplish this, all persons inter- 
viewed were asked what their expectations were about the use of findings, what 
Impact the evaluation had on program operations, planning and policy, and what 
Impact the evaluation had on the program once it was completed. 

Expectations 

It was found that all program managers had positive expectations about the 
eventual Impact of the evaluation on their programs. They expected that the 
evaluation would indicate how the program was working and would lead to future 
Improvements, For example, one manager of a new program expected that the 
evaluation would provide the impetus to bring the program into focus and 
provide direction and thrust for inservlce training. Another expected that 
the evaluation would provide ideas for new program directions to be 
incorporated into subsequent proposals, A further expectation shared by half 
of the program managers was that the evaluation would help to continue or 
expand funding of the program. 

Impact During the Evaluation 

Program managers indicated that sometimes the evaluator provided information 
that signaled problems they could begin correcting while the evaluation was in 
process. For example, during the AEP evaluation, it was found that in some 



schools the criteria for dismissal of students from AEP were not clear. This 
led to clarifying policies and establishing procedures. 



In another project, achieving a revision in the pay scale of the staff was at- 
tributed by the program manager to his interaction with the evaluator. He 
provided the needed encouragement for the program manager to once again re- 
quest a revision in the pay scale and to "push harder" than before. Before 
the completion of the evaluation, the new pay scale was in effect. The proj- 
ect manager considered that the evaluator had had an import iant if somewhat in- 
direct role in this outcome. 

Postevaluation Impact 

Four of the six project managers Interviewed indicated that the evaluation had 
helped give more definition and direction to their programs. The following 
examples were provided by project managers. The AEP evaluation was described 
as helping define goals and objectives. The 1982-83 BTP evaluation was re- 
ported to be the basis for the 1983-84 program. The evaluation of the oral 
fluency of limited English proficient students led to a new secondary school 
level plan, assignment of a Teacher on Special Assignment to work with the 
secondary program, and an increased number of supervisory activities in the 
Bilingual Education program. Finally, the evaluation of the Computer Educa- 
tion Project guided the activities of subsequent years which focused on the 
development of the software consortium. 

A related outcome mentioned by program managers was that evaluations had posi- 
tive effects on program development . They indicated that the evaluation re- 
sults and recommendations provided them with an independent assessment of the 
status of program activities and procedures. In addition, the evaluation pro- 
cess was able to confirm if concerns about programmatic issues were justified, 
and provide suggestions about where modifications might be most effective. 

A frequently mentioned impact of the evaluation process was on program record- 
keeping since evaluations of many programs were affected by a variety of prob- 
lems with records. This led program managers to try to make changes when pos- 
sible. For example, the evaluation of English oral fluency revealed that 
students' records were not up to date, and as a result, new reporting proce- 
dures were developed. The Computer Education project changed procedures for 
keeping requisitions on file duo. to the problems encountered in the evalua- 
tion. The Beginning Teacher Program evaluation confirmed that there were 
problems tracking beginning teachers, and various procedures were instituted 
to improve record-keeping. Also, the evaluation of Project Success (Dropout 
Reduction and Prevention) was affected by difficulty in accessing necessary 
student records, and attempts were made to modify record keeping procedures. 

In keeping with program manager expectations, evaluation reports were found to 
be important for writing proposals, requesting and receiving additional fund- 
ing, and obtaining new sources of financial aid. For example, the CAP re- 
ceived a grant from the Knight Foundation to determine how to improve the 
fund-raising capability of CAP, Inc. The project coordinator believed that 
the evaluation report contributed to CAP receiving the grant. Also, the Di- 
rector of the Computer Education project indicated that the evaluation find- 
ings contributed to obtaining ECIA, Chapter II funds each year. 
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Another outcoc^e mentioned by two program managers was that the evaluations led 
to their requesting other evaluations on different programs or on related as- 
pects of the evaluated program. This was specifically cited in two of the six 
evaluations under study. For example, the Director of Advanced Academic Pro- 
grams requested a formal evaluation of the Junior High School Gifted Pilot 
Program as a consequence of his experience with the AEP evaluation. Also, the 
evaluation of the oral English fluency of 12th grade ESOL students was 
followed by a request from the Department of Bilingual/Foreign Language 
Education for an evaluation of the criteria used to exit limited English 
proficient students from ESOL programs at all grade levels. 

In summary. It was found that the evaluations had Impact on program develop- 
ment during and after the evaluation, met the expectations of program man- 
agers, and were perceived as having positive benefit for the programs. 

Program Staff Response To The Evaluation Process 

Involvement of Program Managers and Staff 

In five of the six programs included in this sample, program managers indicat- 
ed that they had been actively involved in the evaluation process. Their in- 
volvement included participating in the formulation of evaluation objectives, 
and the planning cf the evaluations. In addition to this direct participa- 
tion, project managers reported frequent contact and productive interaction 
with evaluators during the development of data collection instruments and the 
data gathering phase. 

In the case of the program manager \'.hat reported non-involvement in the eval- 
uation process, the person had been hired after all planning for both the pro- 
gram itself and the evaluation had been completed and appro^'ed by the funding 
agency. Therefore, there had been program input into the evaluation design, 
but not by the person who was subsequently assigned the responsibility of 
carrying out the program. In ^succeeding years, the program manager reported 
the same kind of involvement as the other program managers. 

Program managers reported limited contact with the evaluator and no input 
during the data interpretation and report-writing phase of the evaluation. 

One of the six project managers reported that he had reviewed evaluation find- 
ings and recommendations with the evaluator prior to the completion of the 
draft of the evaluation report. 

It^was found that school level personnel who were implementing the program had 
little contact with evaluators and little input into the evaluation process at 
any stage. This was reported by program managers and evaluators and confirmed 
in telephone interviews with program staff of one project. 

Program managers and evaluators were in complete agreement that active partic- 
ipation of program administrators in the evaluation process was essential to 
its utility and that it ensured that the evaluation focused on relevant issues 
for program development. They expressed their approval of present procedures, 
but also had recommendations for expanding the role of program staff involve- 
ment in the evaluation process. 



Respondents^ Recommendations 



The recommendations made by program staff called for an expansion of current 
OEA activities. They involve more evaluator contact with school-level staff 
during the evaluation, and with program administrators during the data 
Interpretation and report writing phases. 

Program Implementer Involvement . Various program managers recommended that 
the evaluator should have more contact with program implementers at the school 
level. It was suggested that more contact would increase their interest in 
the findings, motivate them to use the evaluation information for program im- 
provement, and help reduce some of the anxiety that people feel when their 
program is being evaluated. Also it was thought that increased interaction 
with the program implementers would provide the evaluator with more knowledge 
of actual program functioning which would help in the interpretation of data 
and the development of recommendations. 

It was suggested that at the beginning of an evaluation the evaluator should 
meet with all involved staff to provide an overview of the evaluation. It was 
indicated that staff should be informed of the rationale for evaluations, what 
to expect in terms of how the evaluation would be carried out, the kinds of 
records and information that would be needed, what help they could expect to 
receive while the evaluation was in progress, and the kind of information that 
would be available to them at the end of the evaluation. 

In summary, it was recommended that school-level personnel be given a thorough 
orientation prior to the initiation of the evaluation, and that the evaluator 
should maintain contact with school-level staff during the evaluation. These 
two recommendations by program managers were proposed as ways of increasing 
school-level staff interest in, and use of evaluation information, and provid- 
ing the evaluator with greater opportunity to understand how the program is 
actually being delivered. 

Review of Findings and Recommendations . Program managers in half of the pro- 
grams indicated that they would like to be able to review the findings and 
recommendations with the evaluator during the preparation of the report. Both 
evaluators and program managers said that they would benefit from meetings 
while the report was being prepared and/or at the time that the draft re- 
port was sent for review by program staff. They expressed the opinion that 
printed information sometimes was not sufficient for understanding how the 
evaluator arrived at conclusions, and they suggested that an oral presentation 
would help clarify issues. 

It was proposed that the program staff who had worked with the evaluator in 
planning the evaluation should meet with the evaluator prior to the completion 
of the draft report. One suggested time was when the evaluator knows what the 
findings are and has developed his ideas about the recommendations. The in- 
tent of the meeting, according to one program manager, "would not be to change 
the mind of the evaluator but to give the program point of view." Another po- 
tential benefit mentioned by the same program manager was that program people 
would probably feel freer to ask questions and comment on the results before a 
complete draft of the report was presented. This might lead to greater use of 
the evaluation information by program staff. 
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An alternative time suggested for a meeting was during the time reserved for 
the draft review by program staff. A meeting at this time would ensure that 
program staff understood the evaluation findings, and were able to read the 
report critically. This would help in their interpretation of the draft and 
their preparation of the technical review and/or response to be included with 
the published report. 

Among the six programs in this sample, the program managers reported that no 
meetings of the kind described above were held. However one program manager 
indicated that he had discussed the evaluation findings informally with the 
evaluator; a procedure that they both found beneficial. During the draft 
review period, a meeting may be initiated at the request of the program 
manager or OEA. This is an optional procedure within the evaluation process 
and meetings between program and OEA staff are ordinarily determined by the 
specific conditions in each evaluation. 

In summary, program managers indicated that they would like more contact with 
the evaluator during the report writing phase. In particular, they would like 
the opportunity to review findings and recommendations with the evaluator and 
to provide interpretations from a programmatic viewpoint. Maintaining the 
independence and objectivity of the evaluator while involving program staff is 
an issue that must be dealt with if this recoimnendation is implemented. This 
will be discussed in the Conclusions section. 

Timeliness . For those evaluations where reports could not be completed in 
time to write proposals, and plan budgets and programs, it was recoimnended 
that the evaluator hold an interim meeting to discuss initial findings and 
possible program difficulties. This information, although preliminary, could 
be built into proposals and planning for the next school year. A related 
suggestion was that an interim status report with preliminary findings be 
provided to program managers that could serve as a working document for 
planning. This would be a mere detailed report than the current status 
reports now prepared each spring. 

Program managers from two Chapter II programs indicated that final evaluation 
reports generally were not available early enough to aid them in preparing 
proposals for refunding the program the next school year. (Since Chapter II 
proposals must be prepared by April, an evaluation that includes post test 
results, or end-of-school year data, such as grades, attendance records, or 
Spring achievement scores cannot possibly be available at the time the pro- 
posal is being prepared.) Program managers resolved the issue by incorporat- 
ing results from the previous yearns evaluation into the proposal. Also, the 
evaluator, upon request, provided technical input for the proposal. Both 
Chapter II programs in this sample have been continually refunded so that the 
timeliness of the reports has not been a factor impeding project continuation. 

In summary, although the funding cycle for Chapter II and the DCPS budget and 
planning cycle do not parallel the evaluation cycle (especially when end-of- 
the-year data are needed), project managers did not mention timeliness as a 
serious problem inhibiting the usefulness of evaluations. However, an interim 
report with pi ' binary findings presented orally or in written form, if there 
were time, woulc >e of help to program managers and other decision makers, and 
most likely, enh ce the utility of the evaluation. 




Evaluation of New Programs , Managers from two programs (both in their Initial 
year of operation) suggested that the evalitatlon reports provide more informa- 
tion about the context in which the program is carried out. In both cases, 
they expressed the opinion that the evaluation of a new program should be on a 
small scale and directed at assessing implementation rather than outcomes. 
This recommendation needs to be considered jointly with program administra- 
tors. Even thougli OEA has Input into the evaluation plan of proposed new pro- 
grams, it is program staff that have the principal responsibility for the 
Identification of key variables and the design of the new program that will be 
evaluated. Also, the scale of an evaluation may be determined by Federal/- 
State requirements for given programs. 

In summary, program administrators and staff have a favorable opinion of the 
OEA evaluation process and their involvement in it. The recommendations indi- 
cate however that program managers would like to have more contact with OEA 
evaluators If possible, during the report writing phase. In particular during 
this latter phase they would like to be m^jie informed of the data interpreta- 
tion and recommendation development aspects of the evaluation. All of the rec- 
ommendations by program managers reported in this section imply additional ac- 
tivities and meetings that will increase the evaluator's workload and will 
lengthen the evaluation process and extend the amount of time it takes to 
produce a final evaluation report. 

Dissemination and Follow-up Procedures 

Dissemination Procedures 

At present, all final evaluation reports are transmitted to the Dade County 
School Board via the Superintendent of Schools. Copies are also provided to 
all members of the Superintendent's Executive Council and to the program man- 
agers and their supervisors. In addition, copies of thi reports are sent to 
the Department of Citizen Information for general distribution. Dissemination 
of reports beyond this level (e.g., to principals, teachers, DOE, etc.) by OEA 
is determined on an individual basis according to the nature cf the report. 
Program personnel, however, always have the option of extending the dissemina- 
tion activities beyond the level provided by OEA. 

Among the six evaluations in this follow-up study, the OEA dissemination 
procedures described above were followed, and evaluation reports were ttrans- 
mitted to all senior level decision makers. The procedures varied for the 
dissemination of evaluation reports to building-level program personnel. 
Evaluation reports of two programs were not distributed to building-level 
program personnel. Among the four remaining programs, the program managers 
alone or in conjunction with OEA disseminated reports to teachers, principals, 
and other program implementers. For example, in the College Assistance 
Program and the Academic Excellence Program, the managers distributed reports 
to staff during a workshop. All principals with the CAP program received 
reports while principals with AEP programs received reports on request. For 
the Dropout Prevention and Reduction Project, one of the program coordinators 
distributed reports to each school team. 

Dissemination of the report of the evaluation of the English oral language 
proficiency of ESOL students was a joint effort. OEA sent reports to the 
principals of all senior high schools, and the program manager distributed 
copies of the executive summary to all ESOL teachers participating in an in- 
s-^rvice course in the Fall of 1984. 
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It was beyond the scope of this study to determine the Impact of the dissemi- 
nation of evaluation reports to staff members of the four programs. However, 
in the follow-up telephone survey of a random sample of eight AEP schools con- 
ducted eight months after the workshop, it was found that only two teachers 
had copies of the repcrt although all had attended the workshop where the 
reports were distributed. The other teachers reported that they had not re- 
ceived a copy. When the teachers were asked about the recommendations of the 
evaluation, four teachers (including the two with reports) were able to 
identify one of the major evaluation recommendations. They indicated that 
their knowledge of the evaluation recommendation was based on discussion in 
the workshops, and also on information provided during follow-up meetings with 
area directors and other AEP teachers. 

These findings suggest that provision of evaluation reports may be an 
ineffective way to disseminate evaluation findings unless other kinds of 
follow-up activities are also planned. In addition, the cost of printing 
evaluation reports lends further support to the need to consider other 
dissemination strategies. For example, the average printing cost of the six 
different evaluation reports included in this study was $2.70 and ranged from 
$1.00 (17 page report) to $7.50 (108 page report). 

In summary, in this follow-up study it was found that OEA always transmitted 
evaluation reports to senior level decision makers and program managers. 
Also, OEA dissemination of evaluation reports to school-level personnel 
occurred on a program- by-program basis; for example, OEA distributed the 
report of the evaluation of English oral language proficiency of ESOL students 
to all senior high principals. In addition, some program managers assumed 
responsibility for disseminating full reports or executive summaries to 
school-level staff. This indicates that program managers perceived the 
reports as providing useful information that should be shared with those 
responsible for carrying out the programs on a day-to-day basis. However, the 
effectiveness of disseminating reports to all involved personnel without fol- 
low-up activities needs to be assessed further. 

Follow-up Procedures 

Follow-up contact between evaluator and program manager for discussion and 
clarification of the contents of the evaluation report occured infrequently 
and usually was initiated by the program administrator. Follow-up contact 
between the evaluator and school-level program staff was reported for only one 
project. 

Program managers from three of the projects indicated that they had met with 
an OEA evaluator after dissemination of the report to discuss the findings and 
the recommenationS:> These program managers indicated their s^atisf action with 
current procedures since they always had access to information when they 
needed it. They also indicated that the OEA evaluator had always been avail- 
able to explain technical issues when they requested clarification. 

Of the three projects where follow-up contact was not reported, two were 
evaluated during the next school year. Both program managers indicated that a 
specific meeting had not been needed; however, both thought that for other 
projects it would be a good policy. One program manager reported no follow-up 
contact with the evaluator. He indicated that such a meeting would have been 
helpful because it would have given him more perspective on his program. This 
was a project for which there was no scheduled evaluation during the next 
school year, and therefore no cont-'-^uing contact with OEA. 
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It was found that occasionally the meaning of a recommendation or findinjg, or 
the procedure for implementation was not clear to a program manager, but clar- 
ification was not requested. This occured even though the program managers 
had Indicated satisfaction with their involvement in the evaluation process 
and their relationship with the evaluator. This suggests the need for some 
formal follov-up mechanism to ensure that evaluation findings and recommenda- 
tions are understood by program managers. 

Respondents ' Recommendations 

All program managers agreed that it is important to disseminate evaluation 
findings and recommendations to every person involved in the program. While 
some program managers recommended that evaluation reports be distributed to 
each individual, others suggested that summaries of the evaluations in a non- 
technical style be prepared for distribution to all program staff. 

Program managers also indicated that an oral presentation of findings and rec- 
ommendations by the evaluator to the program staff would help ensure the even- 
tual utilization of evaluation information. As indicated above, program man- 
agers took it upon themselves in four programs to distribute reports, or summa- 
ries, and to discuss the evaluation briefly with program staff. However, 
these same program managers recommended that the evaluator make oral presenta- 
tions to staff. This would provide opportunities to clarify issues as well as 
to establish a feedback process between evaluator and staff to facilitate fol- 
low-up. The majority of evaluators also commented that they would like the 
opportunity to discuss their evaluation with program managers and staff. 

Recommendations from program managers indicated that they would like to see 
OEA adopt a policy of more active dissemination and follow-up of completed 
evaluations. This was reflected in suggestions for more personal contact and 
discussion with the evaluator once the report is disseminated, and in the sug- 
gestion that evaluators make presentations at staff workshops and district- 
wide conferences. It was also reflected in suggestions that OEA prepare docu- 
mentation about the evaluation in a variety of written formats in addition to 
t:he usual full report and executive summary currently disseminated by OEA. 

Various types of summary forms were suggested including: 1) a nun-technical 
report destined for teachers and staff involved in carrying out the program; 
2) summary document to principals presenting recommendations and suggested ac- 
tions; 3) periodic research updates briefly describing the salient findings of 
each completed evaluation so that the field would understand the findings and 
their relevance for the evaluated program; 4) summary of evaluation reports to 
be included as part of the Budget Review process, and 5) annual written and 
oral presentation of evaluations at a School Board Conference Session. 

In summary, the recommendations by program staff all imply OEA undertaking ad- 
ditional activities. In order to satisfy the different needs of all the in- 
terested audiences, various kinds of written materials were suggested. Also 
highlighted was the need to make oral presentations to explain the evaluation 
findings. These recommendations indicate that the program managers were 
pleased with the evaluation process and want to expand it to enhance their 
understanding and that of their staff. To effectively implement the suggested 
activities would increase the workload of each evaluator. Given the present 
responsibilities and evaluation demands, careful consideration and experimen- 
tation would be needed to determine which, if any, recommendations could be 
implemented under present conditions, and which would require additional funds 
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or significant tciiijustiaents in work schedules. 

DISCUSSION AND CONCLUSIONS 

To vhat extent vere recommendations addressed by program managerg f 

It was found that the majority, 84Z of recommendations, vere Is^PX^^et^c^d or 
were in the process of being implemented within a year of the (j^^P^e^iou of 
the evaluation. All the recommendations had been reviewed b;y ^ti^ |>foj^ct 
managers and consideration had been given to the feasibility of ^XnVytji^tittng 
each of the recommendations. These findings indicate that {t^^ Pade 

County Public Schools evaluation Information is utilized by progt"^ \^^^^gets. 
This is consistent with other studies in the evaluation utilizatlc;^ l^t^f^ture 
that indicate that there is a greater tendency at the local level ^^C upon 
evaluation information than at the state or national level (Alklp^ ^J^^l^A ^nd 
White, 1979; Boruch and Cordray, 1980; Caplan, Morrison and Staa^^^^t^^K 1975; 
King and Pechman, 1984; Weiss, 1981). 

What factors influenced the utilizatlou of program evaluation^ ^jv^jlj^gs and 
recommendations? 



All the recommendations that ha^\ been fully Implemented (includl^^ ^^th those 
that suggested new actions, and those that suggested intensified^ i^^ o£ ac- 
tions) did not require significant financial expenditures or new ^^^^et: dislo- 
cations. In general. Implementation of these recommendations coul^ c^^^^ied 
out by the program manager and staff. The key to implementatioti v^^^ C^e pro- 
ject manager's agreement with the recommendations and commitment ^E^l^ijien'- 
tation* These findings are consistent with information about ^v^^^^t^o^ Re- 
ported elsewhere (Leviton and Hughes, 1981; Patton, Grimes, Guthr^^^ ^^eAU^Q^ 
French and Blyth, 1977). 

The recommendations that vere partially implemented or were not l^fX^V^t^d by 
the time of this follow-up are illustrative of major factors ^h^x a^f^ct 
utilization of evaluation recommendations. These includes ^i^aUc^al 
constraints; misinterpretation of findings and recommendations; a^^ti^^t^zat^on 
from higher level DCPS decision makers; and action by individual ^^ti^^l^^^XeVel 
administrators. Another factor influencing implementation is d^cld^on 
maker *s desire for more information before taking action. This ^C^^^^^ ^or 
example, when there is a series of evaluations planned for a prog/^^ the 
decision maker decides that action will be taken only if consl^i^^tit t^eBu^ts 
are found over a period of years. 

Those recommendations that required a sufficient change in prog):^ ^i^aAc^ng 
for their Implementation were less likely to be implemented within ^ ^^^r* In 
each instance, the project manager, and the director with a^^t^^^^t^tlve 
responsibility over the program was in agreement; however, budget ^^^^^atious 
were not made, at least within one year of the completion of the e^^^^^^l^n* 

Another factor that influenced partial Implementation was the de^^^e i^o ^hich 
persons other than the program manager were responsible for the Iti^^^^tvt^tion 
of a recommendation. In the majority of these instances « it was ^^t^C^i^*^ that 
implementation had occurred In only some schools. 

The maj ority of recommendations tended to focus on general ar^^^ 0^ ^e^d^d 
action 9 leaving specific procedures and strategies for implement^!; f^f the 
program staff to design. The high percentage of implementat^^ti ^^ri be 
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attributed » in part, to the form of the recommendations (general rather than 
specific) 9 and to the fact that the majority called for the intensification, 
enhancement or expansion of existing procedures, strategies and policies. 

What kinds of impact did the evaluations have in addition to any changes in 
progrM_brought about by the implementation of evaluation recommendations ? 

Evaluations had various kinds of impact. Among those described by the program 
managers were: 1) Evaluations helped improve the definition, direction and 
scope of activities of the program, and provided confirmation of those 
procedures and activities that had been in place and were effective. 2) 
Evaluations Identified problems in record keeping which led to tightening up 
of record management procedures. 3) Evaluations provided support for contin- 
ued funding, and for obtaining funds from new sources. 4) Evaluations identi- 
fied program needs that required additional study. This resulted in requests 
to OEA for new evaluations, or for the inclusion of new evaluation questions 
in already scheduled evaluations. Also, program manager satisfaction with the 
evaluation process and the kinds of help provided by OEA have led to requests 
for evaluation of programs that had not been evaluated previously. 

In addition, information about program difficulties or needs provided by the 
evaluator while the evaluation was in process had a positive impact. This 
kind of feedback provided program managers with the opportunity to effect 
beneficial changes in program operation within the school year rather than 
wait for publication of the final report. 

What kind of involvement did program administrators and staff have in the 
evaluation process ? 

Program managers were actively involved in the evaluation process. They 
participated in the formulation of evaluation objectives, the planning of the 
evaluation, the development of data collection instruments j and the gathering 
of data. This follows the generally accepted guidelines for carrying out 
evaluations that are utilized by program managers (Brown and Braskamp, 1980; 
Brinkerhoff, Brethower, Hluchyj, Nowarkowski, 1983; Patton, 1978). 

In general, program managers were not involved in analyzing evaluation data^i 
interpreting the results, or formulating recommendations « They did review the 
final draft of the evaluation report and, at that time, provided corrections 
for "errors of fact" when found, and expressed opinions regarding interpreta- 
tions of the evaluation data and, in some instances, implementation of recom- 
mendations. 

School- level program staff received little orientation about the evaluation 
process, had minimal input into the design of the evaluation, and did not 
participate in the review of findings and recommendations. Therefore, the 
persons directly involved in carrying out the programs, and often the ones 
responsible for providing data basic to the evaluation were unlikely to have a 
complete understanding of the evaluation process. This could impact on their 
participation in the evaluation and their motivation to carry out recommended 
program changes. 

What evaluation procedures might be included to enhance evaluation 
utilization? 

Nearly all the people interviewed indicated that the utility and impact of the 
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evaluation informawion cowld be enhanced by increased attention to three 
factors: relevance » participation and timings Increased dialogue between 
evaluators and program implementers (especially principals and teachers) to 
comsmnlcate the purposes, methods and res^ilts of the evalu::tlon could have an 
idipact on Implementation of recommendations « Also* contact could produce 
useful input from implementers to evaluators. This communication should have 
the effect of increasing the relevance of evaluations to the people who are 
most immediately affected by it, thus enhancing their responsiveness to the 
recommendations • 

Program managers, evaluators and nationally recognized experts on evaluation 
utilization believe that increasing the participation of progr^ administra- 
tors in the interpretation of evaluation findings would enhance utilization of 
information and implementation of recommendations (Alkin, et al, 1979; King 
and Pechman, 1984; Patton, 1978). Also, reviewing evaluation findings and 
interpretations during the report writing phase could help to ensure that the 
final evaluation report adequately reflected the program ccntext In which the 
evaluation was conducted. Participation, particularly by program managers, 
could be viewed simply as the first step in the utilization process with pro- 
gram people 2issisting in determining hov evaluation findings might best be 
used to improve programs. 

While program input is important to the evaluator when verifying 
interpretations of findings and considering recommendations, it also is 
essential that the iudependence and objectivity of the evaluator not be 
compromised. Therefore, an important issue is to establish the timing and the 
procedures for review of evaluation results by program staff* It is advisable 
that meetings should only be held after the data have been analyzed, and 
interpretations and recommendations have been written. With this preliminary 
documentation as a frame of reference, the interpretation and implications of 
the findings can be discussed more readily. A policy could be established to 
determine how unresolved differences in interpretation will be reported. 

Because of unavoidable scheduling differences between proffr^»'» {irtd budget pre- 
paration deadlines and the evaluation errd-ol-year-measurement needs in most 
projects, it was recommended that OEA evaluators produce interim summaries of 
preliminary findings on an as-needed basis. For maximum impact and usefulness 
of such preliminary findings, it was suggested by those interviewed that the 
findings be presented orally in meetings, and if time permitted, in written 
reports as well. 

Are current dissemination and follow-up procedures effective for ensuring that 
program staff understand findings and recommendations ? 

The current dissemination procedure, in which OEA distributes reports to 
program managers and they and/or OEA (on a program-by-program basis) 
distribute the same report to school-level program implementers, appears to 
have limited effectiveness. It was found that reliance on this method of 
dissemination did not produce, among program implementers , the level of 
awareness and knowledge of evaluation results that might have been expected 
given the quantity of reports distributed. 

When evaluation results were remembered by implementers, it was usually 
associated with a discussion by a program director or supervisor in a workshop 
or supervisory meeting. Thus program administrators were influential in the 
dissemination of evaluation information and their leadership was essential to 
the staff carrying out the recommendations. 
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Generally, follow-up contacts betveen evaluat'^^^r and program managers vere usu- 
ally Initiated by program staff and vere limited in both number and scope. 
Even vhen clarification of some aspects of the evaluation report was needed, 
some program administrators did not request a meeting or even an explanation 
by telephone. 

What dissemination and follow-up procedures might enhance implementation of 
recomg)endatlons? 

It was agreed by those interviewed that OEA should undertake a broader range 
of dissemination activities to be sure that all program participants clearly 
understand results and their importance. It was recommended that 
dissemination activities include both oral and written presentations^ Oral 
presentations T'ere strongly urged in order to clarify my doubts arising from 
the written presentation. Also, it was indicated that there was ^ need to 
regularly schedule a follow-up meeting between the evaluator and the program 
manpger at the completion of each evaluation to discuss findings and the 
feasibility of implementing recommendations. 

A variety of written formats were suggested for disseminating evaluation 
information, including: reports written in non-technical style; summaries 
highlighting recommendations anu suggested actions; periodic research updates 
distributed district-wide describing the salient findings of each completed 
evaluation; and documentation of, all evaluations in process to be presented at 
a Scliool Board Conference Session. 

The adoption of any or all of these dissemination and follow-up activities 
imply a set of additional tasks for the present OEA program evaluation staff. 
This would impact on the number of evaluations that each evaluator could 
undertake in a school year. In addition there are cost implications related 
to the printing and distribution of each of the different kin^3 of reports. 
However, the recommendations of program administrators and implementers , 
familiar with OEA evaluations, indicate that wider dissemination of evaluation 
findings would provide useful information to DCPS staff who are currently 
unfamiliar with the scoi 3 and depth of the evaluation projects carried out 
annually. 

RECOMMENDATIONS 

The recommendations are based on the findings and conclusions of this 
follow-up study and the suggestions of OEA evaluators and DCPS program 
administrators. They also incorporate concepts and successful procedures and 
strategies that have been developed by evaluation specialists involved in 
federal, state and local evaluation. These recommendations were prepared with 
the objective of increasing the effectiveness and the impact of evaluation on 
program development in the Dade County Public Schools. 

1. Institute procedures to routinely follow-up all major evaluations after 
dissemination of the final report . Evaluation plans of programs 
scheduled annually for evaluation should include an assessment of how 
information and recommeadations from the previous evaluation were 
utilized. This is a requirement of the State of Florida Beginning 
Teacher Program and it has been successful in DCPS. Different follow-up 
procedures will be needed for programs that are not evaluated on a yearly 
basis. 
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2. Involve program administrators In a preliminary revlev of the findings 
and recommendations . This would provide the evaluator with program Input 
to ensure that interpretations are based on technically correct 
information and are programatically meaningful. Procedures would need to 
be considered to determine the appropriate time to hold the review » and 
the documentation to be presented* 

3. Offer program Implementers more extensive and earlier orientation and 
opportunity for input into the evaluation process . A meeting should be 
scheduled with school-level program staff to orient them to the 
evaluation » to obtain their perspectives on the functioning of the 
program, and to identify questions that need to be addressed. After the 
report is published, an oral presentation of findings and recommendations 
should be made by the evaluator with the program manager. 

^* Increase the number and kinds of dissemination activities undertaken by 
OEA. It would be useful to develop different kinds of written formats 
appropriate to the needs and interest of the various audiences involved 
with the evaluation. Oral presentations should also be planned. In 
addition, consideration should be given to the responsibilities of 
program administrators in the dissemination process. 

5* Provide for each participating school a summary of the data collected at 
that school . This feedback to each individual school would provide 
useful information for improving activities, and would motivate staff to 
participate in future evaluations, 

6. Consider establishing a policy for preparing recotmnendatlons by OEA 
evaluators that incorporates criteria which have proven effective . 
Criteria to be considered include: difficulty of implementation, 
specificity of suggested actions, and the potential cost. 

7. Incorporate into the proposed OEA 1985-86 follow-up study an analysis of 
how different levels of administrative and program staff address 
evaluation recommendations and utilize evaluation information . This 
study would be designed to track how evaluation findings are used and 
recommendations are addressed beginning with the program staff review of 
the evaluation draft. It would also examine the relation of 
dissemination strategies to decisions and actions. 

Adoption of these recommendations would result in an expansion of OEA program 
evaluation staff activities, and increased costs for carrying out evaluations. 
This would impact on the number of evaluations that OEA could undertake with 
its present resources. 
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DESCRIPTION OF THE SIX SELECTED EVALUATIONS 



College Assistance Program (CAP) 

This program is a component of the College Admissions Service Office which is 
in the Division of Student Services. It was established in 1977 to encourage 
and assist the efforts of high school seniors to pursue post-secondary educa- 
tion. CAP advisors provide assistance to students sesklng 1) financial aid, 
2) information about colleges and entrance examination, and 3) help filling 
out applications. In addition, as part of the program, a scholarship fund was 
set up to be administered by CAP, INC., for needy students whose post-secon- 
dary education financial aid packages were insufficient. 

In the 1977-78 school year, CA? was implemented on a limited basis . As of the 
1982-83 school year (the. year the evaluation was conducted), a total of 35 
part-time CAP advisors were located in all 24 senior high schools in Dade 
County. 

The evaluation was designed to examine the extent to which the program was 1) 
meeting its goals and 2) providing a worthwhile service. The evaluation was 
conducted by means of 1) surveys distributed to administrators and guidance 
personnel in the senior high schools; 2) interviews with administrators, gui- 
dance personnel, and students; 3) data collected for The Placement and Fol<» 
low-up Reports by the Office of Student Support Programs; and 4) data supplied 
by the College Admissions Services Office. 

Academic Excellence Program (AEP) 

The Academic Excellence Program (AEP) is a component of the Department of Ad- 
vanced Academic Education. It was established in 1983 to provide an enrich- 
ment curriculum for above average students in grades K-6 and to assist them in 
maximizing their Intellectual potential. 

In the 1983-84 school year, AEP was initiated in 24 schools with a total of 28 
teachers participating. In the majority of schools, AEF classes are scheduled 
during the school day, but in a few the classes are all after school. 

The evaluation of this program was designed to examine the process of program 
development and initial implementation during its first year. The evaluation 
was conducted by means of (1) a review of student participant rosters, (2) on- 
site observations of program activities > (3) survey instruments completed by 
parents, students, administrators ^ AEP teachers, and regular classroom teach- 
ers, and (4) interviews with program personnel. 

English Oral Fluency of Limited English Proficient (LEP) Twelfth Grade Stu- 
dents (ESOL) 

The DCPS Board wanted to know to what extent LEPs were graduating without Eng- 
lish fluency. There was concern that LEP students were graduating without the 
capacity to function effectively on the job or in college because of their in- 
ability to express themselves adequately in English. Therefore, ITie Dade 



County Public School Board directed staff to evaluate the oral proficiency of 
twelfth grade LEP students eligible to graduate. This Is the only evaluation 
In the sample that focused solely on determining one program outcome, oral 
fluency In English. 

Students who are non-native speakers of English, and who lack proficiency In 
English (limited English proficient or LEP) are enrolled In English for Speak- 
ers of Other Languages (ESOL) classes until such time as they can demonstrate 
competency in English. ESOL classes are provided in all senior high schools 
with a LEP population by trained and certified ESOL teachers. Students enroll 
in ESOL while taking content subjects to meet basic course requirements. Eng- 
lish oral proficiency is not a state or district requirement; students may 
graduate while still enrolled in ESOL. 

The evaluation was conducted by testing the oral proficiency of LEP students 
and students who had exited ESOL within the last twelve months. Independent 
oral proficiency rating:? were obtained for students from their respective ESOL 
or regular English teacher. 

Dropout Prevention and Reduction Project (Project Success) 

The Dropout Prevention and Reduction Project (Project Success) began in 
1981-82 school year as a component of the Division of Student Services to pre- 
vent and reduce dropout among high-risk secondary level students. In 1982-83, 
Project Success received ECIA, Chapter II funding to Implement activities with 
students and a reward system designed to provide incentives for Improved be- 
havf.or and scholarship. 

Duriug 1982-83 the initial year of Chapter II funding. Project Success ope- 
rated in foi^r senior high schools. In each school > the Project was carried 
out by a support team composed of two to four classroom teachers, one adminis- 
trative liaison, a counselor, an occupational/placement specialist, and a 
visiting teacher* Area Intergroup Relations Specialists collaborated with and 
directed the support teams; one specialldt had responsibility for coordinating 
the activities of the four schools and the program evaluation. Funding was 
renewed for 1983-84. In 1984-85, Project Success and two other dropout pre- 
vention initiatives were combined into one. Project Success, and funded with 
Chapter II monies. 

The 1982-83 evaluation was designed to assess the effectiveness of the program 
on student school performance and behavior and to determine school staff and 
students* perceptions of the project *s Impact. The evaluation was conducted 
by means of (1) examination of Grade Point Average (CPA), attendance and con*- 
duct records of participating students; and (2) surveys of administrators, 
teachers and project students in the four schools to determine the Impact of 
the project, and its strengths and weaknesses. 

Computer Education Project (CEP) 

The Computer Education Project (CEP) was established with a grant from Chaptet 
II funds in 1982. The Department of Basic Skills began the Computer Education 
Program in 1980, and in two years had acquired more than 600 computer systems 
located in 132 schools. The CEP was designed to provide support and Increase 
the utilization of these computer systems by (1) providing for the maintenance 
and enhancsment of the existing computers; (2) establishing a software consor- 
tium. Including software and equipment, staffed by a full-time media special- 
ists; and (3) acquiring additional microcomputer equipment. 




The 1982-83 school year was the initial year that Chapter II provided funds 
for the CEP, Chapter II funds have beec renewed yearly. 

The 1982-83 evaluation was conducted by means of (1) the examination of 
school inventory lists, purchase requisitions and orders provided by the 
Department of Basic Skills, Division of Budget and the Purchasing Department; 

(2) a telephone survey of computer education contact persons; and (3) 
interviews with computer education specialists and media specialists. 

The Beginning Teacher Program (BTP) 

The Beginning Teacher Program (BTP) was established in 1982 by the State of 
Florida in an -i^fort to Improve the quality of its educational systems. 
Teacher participation was mandated in a year long Beginning Teacher Program as 
a requisite for regular certification for beginning teachers. The program is 
designed to provide each beginning teacher with a supervised support system to 
maximize professional competence on twenty-three essential teaching skills . 
The BTP is a component of the Bureau of Staff Development, but depends on the 
Bureau of Personnel Management for initial identification of potential parti- 
cipants. 

In the 1982-83 school year, the BTP was initiated and Implemented district- 
wide. By state Isw, each beginning teacher was assigned a support team com- 
posed of the school principal, a designated peer teacher, and another profes- 
sional educator ^ usually an area or central office administrator, or a univer- 
sity professor. 

The evaluation of the Beginning Teacher Program's first year of operation was 
designed to examine (1) the appropriateness and effectiveness of major program 
elements, (2) compliance of activities with district and state guidelines, and 

(3) program impact on beginning teacher performance on the twenty-three essen- 
tial teaching competencies. The evaluation was conducted by means of (1) sur- 
veys completed by beginning teachers and support team members > (2) interviews 
with selected program participants, and (3) review of program documentation. 
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TABLE 3 

Iiplementfltion Status of Reconmendatlons from Six Evaluation Reports 



Project 



Publlcat'on Kecoomendatlon 
Date 



Status of Reconendatlon Coments and Clarification 



College 
Assistance 
Progrfiffl (CAP) 



October 1984 



3o 



1. Increase funding to 
provide full-time, five 
days a week coverage in 
those schools that need 
extended service. 



2, Initiate a review of 
the classification of the 
CAP advisor position. 



3. Increase the outreach 
and publicity activities 
of the CAP advisors to 
encourage wider student 
knowledge and use of CAP. 



4. Encourage CAP advisors 
to begin more intensive 
work with students earlier 
in the eleventh grade - 



Partially Implementid. 
Request for full-tiioe CAP 
advisors was Included in 
1984-85 Department of Stu- 
dent Services proposed 
budget. 



Implemented. Recomiaenda- 
tlon ccnfimied actions 
taken by program manager. 



Partially Implemented. 
Recommendation reinforced 
program direction. 



Implemented as presented. 
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In 1984-85 four senior high 
schools expanded their covertgt 
to full-time, A total of six 
schools have full-tiie coverigi 
because principals pay for thi 
additional days from the school 
budget. 

Program manager requested re- 
classification of the CAP advi- 
sor position during the evilui- 
tion. Interim findings provid- 
ed support for the requett. 

Additional activities were 
undertaken by a few advliori 
and administrators. FiiU ii- 
pleientation depends on budget 
increases that will provide 
more staff time for outreach. 
Also, alternative strategies 
need to be developed. 

During 1984-85, vorkshopi 
oriented CAP advisord, espe- 
cially nev staff, to nore 
effectively use their Um. 
Prccedures facilitated 
attending to eleventh gride 
students earlier in the school 
year. 
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TABLE 3 (Coii"t,) 

Implenentatlon Status of Reeowridations from Six Evaluation Reports 



h'oject PiiUicatlon Rficomendation Status of Recomendation CoMents and Clarification 

Date 



CAP 



3/ 
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3. Encourage CAP advisors 
to conduct assemblies for 
junior high students to 
Increase their awareness 
of post-secondary educa- 
tion possibilities and 
facilitate better course 
planning. 



Implemented b6 prcscnced. 
Program administrators in- 
tensified attention to 
this activity. 



6. Clarify the eligibility 
requirements for the CAP, 
Inc. scholarships with the 
advisors to insure an 
equitable distribution of 
funds. 



Considered and rejected. 
Possible misinterpretation 
of recommendation. 



In 1984-85 there vas an in- 
crease in the number of CAP id'> 
visors included on the senior 
high airticulatlon teams that 
visited junior high schools. 
Due to time limitations, the 
majority of advisors did not 
participate in actlvltlis it 
the junior high level. 
Increased activity depends on 
expanded full-time coverage. 

Program administrators Inter- 
preted recommendation to sug- 
gest possible bias In distribu- 
tion of funds, and disagreed 
with that. However, thi recoi- 
mendatlon vas directed at the 
funding in students not plan- 
nlng to attend four-yaar col- 
leges. Thsy vere not being 
encouraged to apply for scho- 
larships. Program administra- 
tors Indicated that It vas un- 
likely such students vould meet 
eligibility requirements. 

Similar concerns about distri- 
bution of funds vas raised by 
consultants vho studied the 
scholarship program and GAP, 
Inc. fund-raising actlyltias. 
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TABLE 3 (Con't) 

Inplenentatlon Status of Reconmendatlons froi Six Evaluation Reports 



Project 



Publication 
Date 



Recomendation 



Status of Reconendaticn Consents and CUrificatloa 



CAP 



7. Review current training Inpleoented as presented, 
procedures to ascertain 
whether they adequately 
inform new advisors. 



6. Provide adequate office Not lopleoented. This is 
space and privacy for all the responsibility of 



CAP advisors. 



school adninlstration, not 
CAP administration. Im- 
plementation depends upon 
available facilities at 
each school. 



Training procedures are ravlev- 
ed annually. After the report 
was published, another reviev 
was held. Program admlolstra- 
tors agreed that m CAP idvl* 
sors needed more training. Is 
the regularly scheduled vork- 
shops more orientation vii 
given to new advisors. Tine li 
not available to provide suffi- 
cient on-the job training vhlcb 
is needed also. 

Program administrators have ri« 
commended that advisors be pro* 
vided with adequate office 
space. Problem la shared by 
all student services peraonntl 
and little Inproviment la poi* 
slble. 
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TABLE 3 (Con't.) 

Implementation Status of Recommendations from Six Evaluation Reports 



Project hblicatlon Recommendation Status of Recommendation Comments and Clarification 

Date 



Academic September 
Excellence 1984 
Program (AEP) 



1. Information regarding Implemented as presented, 
children's progress in the Recommendation dependent 
program should be more upon program staff 



frequently provided to 
parents 



2. Teachers who teach the 
after-school programs 
should receive equitable 
compensation for extra 
time required. 

3. Program instructional 
staff should be provided 
with additional Insenice 
training related to the 
operation of the program 
and Instructional activi- 
ties. A survey of their 
needs might be made prior 
to the actual provision of 
inservice training. 



(school-site) personnel 
for implementation. 



Implemented as presented. 



Implemented as presented. 



Program director required 
schools to send home periodic 
reports in order to keep pa- 
rents informed. In a survey of 
8 randomly selected schools 
with AEP programs, all had sent 
home semester or quarterly re- 
ports. This vas a 38Z increase 
in the number of schools 
sending home information In 
1983-84. 

The salaries of after-school 
teachers were adjusted and they 
were paid the equivalent of a 
sixth period supplement. 



Three full-day inservice work- 
shops were presented to AEP 
teachers. In two of the four 
areas, additional Inservice vas 
provided through monthly meet- 
ings of the area director vith 
all AEP teachers. 
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TABLE 3 (Con't.) 

InpleiDentatlon Status of Recoomendatlons fron Six Evaluation Reports 



Project 



Publication 
Date 



Recomendation 



Status of Reconmendatlon Connents and Clarification 



AEP 



43 



4. Goals and objectives 
should be established for 
the prograi at the dis- 
trict level that are spe- 
cific enough to enable the 
definition of suitable in- 
struments to assess the 
impact of the program, yet 
sufficiently flexible to 
allow individual schools 
some latitude in accommo- 
dating differences in stu- 
dent population charac- 
teristics and instruc- 
tional capabilities. The 
latter qualification ad- 
dresses the evident reluc- 
tance of many respondents 
to support the notion that 
program curriculum common- 
alities should exist 
across all program schools. 

5. An effort should be 
made to more adequately 
orient parents to the pro- 
gram and more clearly ex- 
plain the admission 
criteria. 



Partially implemented. 



Implemented as presented. 
Recommendation dependent 
upon program staff 
(school-site) personnel 
for implementation. 



Recommendation is in the pro- 
cess of being fully impleunt- 
ed. The prograa director ac- 
tively tried to liplement Chli 
recommendation Id 1984-85. A 
curriculum program vas Idinti- 
fied that meets progria niidi 
and gives acceptable focui for 
the diverse programs. Tliiough 
workshops AEP teachers hm 
come to agree that a conon lat 
of goals and objectives ire 
needed and possible. 



ERIC 



Program director agreed with 
the recommendation and support- 
ed Its Implemintition. AEP 
telephone survey indicated that 
the recommendation vas carried 
out by the schools. Teicheri 
provided more Infomatlon about 
admissions criteria and the AEP 
program In 1984-85. 
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TABLE3 (Con't.) 

Inplenentation Status of Recomendatlons froD Six Evaluation Reports 



Project Mlication Recomnendation Status of Recomnendation Coments and Clarification 

Bate 



^ Letters explaining the progrn 

vere sent Ym by all achooli 
in the suney vhich was a lit 
increase over 1983-84. Alio 
seven of the eight schools, 88Z| 
reported holding an orientation 
meeting with parents. Only 381 
held meetings in l'983-84. 

6. If at all possible, the Under consideration. The decision to hold AEP clai- 
AEP should be scheduled ses only during the regular 

during regular school school day is pending the re- 

hours at all schools. suits of the 1984-85 evaluation 

which is designed to asseii thi 
impact of the program on cogni- 
tive abilities. If it ii found 
that program impact is not ai 
strong for the after-school 
students, then the recommenda- 
tion will be implemented. 
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TABLE 3 (Con't.) 



Project 



Inpleaentatlon Status of Recomendations from Six Evaluation Reports 



Publication 
Date 



Recoimendatlon 



Status of RecomiiendatlGR Conents and Clarification 



English Oral August J 534 1. Future evaluations of 
WuMcy of LEP students in ESOL pro- 

llmited Eng- grams should examine the 

lish Profi- factors which contribute 

cient (LEP) significantly to students' 

Twelfth Grade acquisition of English in 

Students the most effective, rapid 

(ESOL) and cost efficient way. 

Oral testing of Individual 
students appears to be a 
viable method of determin- 
ing oral English fluency, 
and should be one of the 
measures used. 



Implemented as presented. 



The 1984-85 evaluation of cri- 
teria for exiting ESOL is a di- 
rect outcome of the recopcn- 
datlons and findings of tbi 
1983-84 evaluation. The proci- 
dures identified in this recoi' 
mendation are being employed In 
the 1984-85 evaluation. 
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2. Oral proficiency, al- 
though important, is only 
one measure of a student's 
abilities in a second lan- 
guage. A complete assess- 
ment of a student's total 
language abilities should 
address reading, writing 
and cultural dimensions as 
well. Other factors, such 
as grades, exposure/con- 
tact to the second lan- 
guage outside of school, 
motivation and teacher 
Judgment, in the form of a 
"profile" should also be 
included in the long-range 
plan for establishing 
English proficiency cri- 
teria for 12th grade ESOL 
students. 



Under consideration. Im- 
plementation is dependent 
upon the outcome of the 
1964-85 evaluation of ESOL 
exit criteria at the 
senior high school level. 
If implemented it will be 
for all grade levels, not 
only 12th grade. 



The 1984-85 evaluation of cri- 
teria for exiting ESOL iddreii- 
es the issues raised in tbli n- 
commendation. The Bilingual 
Education/Foreign Langutgi 
Department has iddressid thii 
reconendation by initiitlng 
case studies of early and iiti 
exlters. A teacher on ipidal 
assignment (TSA) has been con- 
ducting the case studies. 
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TABLE 3 (Con't.) 

Inpleoientatlon Status of Recomendatlons fron Six Evaluation Reports 



Project 



Publication 
Date 



Reconnendation 



Status of Recoimeadation Coiments and Clirificttlofl 



ESOl 



3. Oral testing of indivi- 
dual students appears to 
be a viable nethod of de- 
termining LEP students' 
oral English fluency, pro- 
vided that the test ade- 
quately samples the oral 
language skills and tasks 
the student needs to 
master, is reasonably easy 
to administer and score, 
and minimum training of 
testers or ESOL teachers 
is required. The B.E.S.T. 
and Supplement, or a lo- 
cally developed and vali- 
dated test that follows 
the E.E.S.T. "model," is 
recctmended. 



Under consideration. Im- 
plementation will depend 
on the outcomes of the 
evaluation of exit 
criteria. 



The recommendation proposed 
testing oral language skills of 
lEP students, Evaluation of 
instruments that measure oral 
proficiency was undertaken ai 
part of the 1964-85 ESOL eval- 
uation of exit criteria. Thi 
Instrument (s) and procedures 
selected will depend on the n* 
suits of t^e evaluation, 
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4. The oral ^est selected 
for county-vide use at the 
senior high school level, 
e.g. placement, exit cri- 
teria, etc., should be 
field-tested with a suffi- 
cient number of students 
in each ol grades 9-12, in 
each of the ESOL clsssifi- 
cations. 



Partially implemented. 



1984-85 evaluation of ESOL exit 
criteria included two tests of 
oral proficiency. Both testi 
were applied to students In ill 
grades and ESOL classifica- 
tions. Further field-testing 
may be necessary, but that vlll 
depend on decisions about which 
Instruments will be used for 
establishing exit criteria. 
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TABLE 3 (Con't.) 

Inpleoentatlon Status of Recom&endatlons from Six Evaluation Reports 



Project 



Publication Recoi&endatlon 
Dace 



Status of Reconiendatlon Conients and Clarification 



ESOl 



Si 



5. The Oral Proficiency 
Rating Scale adapted for 
this evaluation would aid 
ESOL teachers In assessing 
the oral proficiency 
skills of their students. 
Fkirther reflneient and 
field-testing of the 
scale, to reflect diffe- 
rent age levels and currl- 
culuQ content, Is sug- 
gested. 

6. Coticurrent validity of 
the iniitruiDentis) selected 
should be obtained through 
adoinistsring, to a sample 
of the senior high school 
studentg^ A) a ScConu, 
non-structured oral profi- 
ciency test, e.g., the 
OPI; and B) a measure of 
all of the language 
skills: listening, speak- 
ing, reading, writing and 
cultural awareness, 
through general language 
proficiency test, e.g., 
the Pre-TOEFL or the SLEP. 
Tsacher ratings of the 
tisted students' oral lan- 
guage skills should alio 
be obtained fron a sanple 
of experienced secondary 
ESOL tiachirs. 



Partially implenented. 



The Bilingual Education/Foriiga 
Language DeparCoent decided to 
refine the scale already In m 
by ESOL teachers rather than 
adopt a new scale, 



Inplenented as presented. 



This recooBendation wai cirrlid 
out in the 1984-85 ESOL mltif 
tion. , 
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TABLE 3 (Con't.) 

leplenentaclon Ptatus of Recomendatlons froQ Six Evaluation Reports 



Project Publication Recomendation Status of Recomendatlon Coo&ents and Clarlflcition 

Date 



7. Student neiberahlp In 
ESOL programs by ESOl 
classification needs tc be 
updated each senester in 
senior high scMs. In 
gathering Infomatlon for 
the selection of students 
for this evaluation I eval- 
uators found mj students 
whose ESOL levels were not 
current. 



Istpleinented as presented. 



The Bilingual Education/rortlgn 
Language Departnent iitibllihid 
stricter policies, requiring 
that the schools keep ioti ac- 
curate records of studiot tn- 
rolloent in ESOL claiui. 



TABLE 3 (Con't.) 

lapleientatlon Statue of Recomendatlons from Six Evaluation Reports 



Project 



Publication 
Date 



Recomendatlon 



Status of Recomendation Coments and Clarification 



Dropout 
Prevention 
and Reduction 
(Project 
Success) 



February 
1984 
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1. TeaiE leaders from each 1983-84: Partially 
of the schools involved in loplemented, 
the project should be re- 
sponsible for maintaining 
records on participating 
students, extending back • 
to at least the ninth 
grade, including such data 
as grades (both academic 
and conduct), attendance 
figures, conduct records, 
and, if applicable, more 
precise reasons for with- 
drawal from school, 



1984-85: Partially 
implemented, 



2. A control group should 
be established to better 
ascertain the effects of 
the project. 



1983-84: Partially 
implemented, 



1984-85: Implemented as 
presented. 



Team leader responsiialities 
were tightened up, but record 
keeping continued to be a pro- 
blem. This was attributed in 
part to the lack of financial 
incentives and staff time to 
maintain records. 

Although records have improvedi 
obtaining accurate records re- 
mained a problem in 1984-85. 
The Issue of financial incen- 
tive was resolved with payoent 
of a 8i\th period supplement to 
team leaders. 

A comparison group in one 
school was incorporated into 
the project in midyear in ac- 
cordance with this recommenda- 
tion. 

A control group was incorpo- 
rated into the design of Pro- 
ject trio, the successor to 
Project Success. A control 
group was selected at the be- 
ginning of the 1984-85 school 
year. 
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TABLES (Con't.) 

Inplementatlon Status of Reconendatlons from Six Evaluation Reports 



Project Publication Recoimendatlon Status of Recomendatlon Coments and Clarification 

Date 



Project 
Success 
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3. More effort should be 
expended toward the in- 
volvement of project stu- 
dent parents. 



4. Some type of credit or 
other reward needs to be 
considered for the teach- 
ers involved in the pro- 
ject, especially team 
leaders. 



1983-84: Implemented as 
presented, 



1984-85: Implementation 
continued. 



1983-84: Considered and 
rejected by project fund- 
ing source. Evaluation 
recommendations were not 
available when the project 
was approved and funded 
for 1983-84. Recommenda- 
tion confirmed 1983-84 
project proposal request. 



Partially implemented, 



Staff reported that they pro- 
vided more opportunities for 
parental involvement, but thin 
was no increase in parental in* 
volvenent. It was suggested 
that additional funding night 
have helped since there would 
have been available staff to 
work with parents. 

Parental InvolveDcnt actlvltiii 
continued. Parents continue to 
be less involved than project 
staff would like. 

Proposal for 1983-84 included i 
request for financial aid for 
participating staff. Thii vii 
independent of the evaluation 
report recomendatlon. Some 
funds were made available for 
staff who tutored studinti 
only. These funds did not ad* 
dress the issue of paying itiff 
for time devoted to project 
activities. 

Recognition of staff efforts 
were made through presentition 
of awards at end of school 
year. 
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TABLE 3 (Con't) 

Implementation Status of Recommendations from Six Evaluation Reports 



Project 



Publication 
Date 



Recommendation 



Status of Recommendation Comments and ClarlflcatloD 



Project 
Success 



1984-85: Implemented as 
presented. 



Funds were obtained for 6th 
period supplement to pay team 
leaders. In one school » the 
principal budgeted a sixth 
period supplement for tvo addi- 
tional team members. Fundi for 
tutoring were available on a 
limited basis. 



5. If additional monies 
can be found i a full-time 
coordinator should be ac- 
quired to follow up with 
project students. 



1983-84: Considered and 
rejected by project fund- 
ing source. Recommenda- 
tion confirmed 1983-84 
project request. 



1984-85: 
rejected. 



Considered and 



Request for full-time coordina- 
tor Included in 1983-84 propo- 
sal. Not funded, 



Request for a full-time coordi- 
nator was included in 1984-85 
proposal, The position vii not 
funded. Although there ire 
staff supenlilng the profraa 
at present I no one person b«i 
been assigned full-tlee to thit 
program. All supervisory stifl 
have multiple responslblUtiii 
involving various dropout pre- 
vention programs. 
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TABLE 3 (Con't) 

Iiplementatlon Stttus of RecoMiendations from Six Evaluation Reports 



Project 



Publication Recomendatlon 
Date 



Status of Recomendatlon Coments and Clarification 



Project 
Success 
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6. The identification of 1983-84: Partially imple- One junior high school vas In- 
students for this project nented. Recomendatlon eluded in the 1983-84 project 
should be Initiated at the confined actions taken by year due to staff initiative, 



eighth gra{'e level. 



7. Within school comuni- 
cations regarding the ob- 
jectives and activities of 
the project, directed at 
both faculty and students 
should be enhanced. 

8. Individual school ac- 
tion plans should be re- 
evaluated toward the end 
of selecting the most ef- 
fective of these for 
future applications. 



staff prior to publication and available funds, 
of evaluation report, 



1984-85: Inpleniented at 
presented. 



1983-84: Inplenented as 
presented. Recomendatlon 
confined actions taken by 
staff prior to publication 
of evaluation report. 



1984-85 project proposal dtad 
this recomendatilon. It vas 
reported as strengthening tht 
proposal. Project Trio (the 
successor to Project Succiii) 
included six junior high and 
Biddle schools In the fiidir 
patterns of project senior high 
schools. Therefore identlfici- 
tion process is beginning 
earlier. 

Meetings vlth faculty vere held 
especially in those schools 
where attitudes toward project 
were negative. 



1983-84: Partially inple- Review of schools plans vis in 
nented. Confined project ongoing process. Tiani vire 
procedures, encouraged to shire their suc- 

cessful pricticei. 



1984-85: Under 
consideration. 



Each school has Its m lodel. 
Part of the current evaluation 
Is an exaDinitlon of the difff 
rent models, Project Trio ex- 
pects to develop virioui 

OOdlll. i}ri 

be 



TABLE 3 (Con't.) 

Implenentatlon Status of Recomendatlons from Six Evaluation Reports 



Project 



Publication 
Date 



Recomendation 



Status of Recomendation Conents and Clarification 



Conputer December 1. The ECIA Chapter II 
Education 1983 Coinputer Education Project 

(CEP) shojild be refunded for 

another year. 



2. Consideration should be 
given to the possibility 
of placing a major empha- 
sis on the completion and 
operation of the software 
consortium in proposals 
for continuation of this 
project. 



Implemented as presented. 
Funding for the 1983-84 
school year vas provided 
prior to the publication 
of the evaluation. Recon- 
mendation confirmed 
actions taken. 



Implemented as presented. 



The program director indicatid 
that the evaluat.lon reporti 
have been supportive vhen pro- 
posals have been presented to 
the Citizen Review Comlttefi 
and the School Board. Project 
vas refunded In 1984-85 and 
evaluation reports were pre- 
sented. 

The computer lab vas a direct 
development of this reconenda- 
tion. It provided thi major 
focus for the computer aduca- 
tion program in aubaequtnt 
years. 
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TABLE 3 (Con't.) 

Inpleinentadon Status of Recomendatlons from Six Evaluation Reports 



Project 



Publication 
Date 



Recomeni^ation 



Status of Recouendation Conents and Clarification 



Beginning Septenber 
Teacher 1983 
Progra (BTP) 



1. Improve the orientation loplenented as presented, 
programs for peer teachers 
by incorporating training 
in conferencing techniques 
and providing detailed in- 
fonsation on the proce- 
dures and content of the 
teacher observation/eval- 
uation methods. 



Peer teachers vere providid 
necessary training via video* 
tape on conferencing techni- 
ques. Also they received orlia* 
tatlon on the TADS evaluation 
instrument. 



2. Implement a more com- 
prehensive orientation and 

:alnlng program for 
Uding-level adalnlstra- 
" v7 and other profes- 
lo-'di educators, 

3. Inlement a review of 
the citiiunlcation network 
between staffing control 
and the BTP office In an 
effort to identify and 
elimlyte barriers to 
speedy identification of 
beginning teachers. Pro- 
cedure,? for notifying the 
BTP office of status 
changes should also be 
reviewed, 



Implemented as presented. 



Implemented as presented. 



Area-wide orientation/training 
meetings vere held in eich of 
the foui areas for building- 
level adninistritors and other 
professional educators. 



Meetings vere held at the be- 
ginning of 1983-84 school ytir 
to discuai find reiolve person* 
"1 issues related to the Be- 
,.nnlng Teacher Prograa. Hov- 
everi problems vere still bilog 
Identified in 1984-83. 
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TABLE 3 (Con't.) 

Iiplementation Status of Recomendatlons from Six Evaluation Reports 



Project 



Publication 
Date 



Recomendation 



Status of Recomendation Coment; and Clarification 



BIP 



4. Initiate more frequent 
contacts with program par- 
ticipants for the purpose 
of providing information 
and more direction. 

3. Periodically aonitor 
support teams to ensure 
that teams are functioning 
properly. This would in- 
clude a review of portfo- 
lios and verification of 
the existence and appro* 
priateness of written pro- 
fessional development 
plans. 



Implemented as presented. 



Implemented as presented. 



Audits were carried out it eich 
site where a beginning teiehir 
was assigned. A newsletter vii 
distributed periodically to lU 
program participants. 

Onslte audit vai conducted it 
each school and Included rcviw 
of the portfolio of each pro- 
ject participant. BTP idilolf 
tratlve staff was Increaaed to 
1983-84. 
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USE OF EVALUATIONS IN EDUCATION: AN OVERVIEW 



The scientific evaluation of education and social action programs 
beceoae important to the federal government decision-making 
process in the 1960 *s. It was a direct outcome of the Great 
society and the War on Poverty initiatives that provided huge 
federal expenditures for a vide variety of government programs. 
Program evaluation was built into legislative funding to ensure a 
system of accountability, a way of measuring effectiveness, and 
an objecive basis for deciding the future of programs (Fatten, 
1978; Weiss, 1977) . 

As the demand for program evaluation expanded, so did the need to 
understand how evaluation information was used, if at all, in the 
decision-making process at national, state emd local levels. 
Since the late 1960 's evaluation utilization has been widely 
research and even more extensively discussed. The emphasis has 
been on defining and describing the uses of evaluation 
information in the decision-making process and the factors that 
contribute to utiiizatidn. 



How Evaluation Is Used 

Wlxile not very common, evaluation information has been found to 
directly affect decisions about educational programs (Alkin, 
Daillak and White, 1979; King and Pechman, 1984). To show that 
this occurred, there should be clear indications that the 
evaluation information was considered and used in reaching the 
decision, and that it was unlikely such a decision would have 
been reached without the evaluation results. Exeuaples of direct 
influence are: the implementation of recommendations as proposed 
in the evaluation; termination of a progreun based on the outcomes 
of an evaluation; and budget increases based on needs identified 
in the evaluation. 

More frequently it has been found that evaluation results 
influence programs and decision-making in ijidirect_and_gradual 
ways. In the real-world of education and f;ocial services 
decision-making, policy changes are usually not abrupt, but occur 
over time as information builds up and political and economic 
forces converge (Berke, 1983; Weiss, 1977; Wise, 1978). 

Evaluation information may influence decision makers* thinking 
about a program without specific actions being tied directly to 
the information. It has been shown that evaluation information 
may enhance tinderstanding of issues, verify the implementation of 
a program, illuminate the contexts to i^^hich programs operate, 
influence ideas and attitudes ^out a program, and confirm the 
value of ongoing program practices (Boruch and Cordray, 1980; 
Weiss, 1981). 

In this reerard research also indicates that evaluation 
utilization occurs even when it canno t be demonstrated that 
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recommendations led to Implementation, or that the evaluation 
findings led to immediate decisions (Brown and Braskzunp, 1980) . 
This has been the "conceptual use" of evaluation information to 
contrast it with "instrumental use" where findings and 
recommendations are used directly in decision-making. Although a 
somewhat arbitrary distinction, it is nonetheless an important 
one to meike because conceptual use is much more pervasive and 
potentially more consequential in program development and 
policymaking (Patton et al, 1977; Rich, 1977; Kennedy, 1984). 

A reoent summary of findings by the Evaluation Training Project 
(Brinkisrhoff , Brethower, Hluchyj, and Nowarkowski, 1983) reflects 
the current thinking of the majority of researchers studying 
evaluation utilization: 



Recent resezurch (Alkin et al, 1979; Patton, 
1978; Braskamp, Brown anC Newman, 1980) 
reinf orce« th\e notion that evaluation can and 
does have iaqpact beyond being "used" in 
decisions. It can reduce decision-makers* 
xincertainty, make people more aware, reinforce 
policy and create support to neune a few 
additional purposes. Further, this research 
shows that evaluation data can rarely be linked 
directly to a decision. To limit reporting 
purposes to decision-making is to drastically 
limit the potential ability of evaluation, 
(pp. 152-153) 



Factors That Affect Utilization 

Factors influencing program evaluation utilization have teen 
studied to identify those that contribute most to tlie use of 
evaluation information. Among the factors studied 
systematically, one has been singled out continually as wielding 
a powerful influence in decision to accept and implement 
recommendations or to reject or ignore them — what has been called 
the "political" factor (Patton, 1978). 

The fact that actions and decision are the desire result of 
educational program evaluations, and that these actions and 
decision impact on allocation of resources, authority and 
position within an institution, cast^^ evaluation clearly as an 
instrument in changing power relationships. There is little 
doubt that such "political" considerations influence utilization 
of findings and recommendations, and objective, professional 
eyaiuators must be aware of them. Insofar as an evaluator sees 
his or her role not only as a provider of information, but alao 
as a member of an institution with a stake in the institatiori ' s 
productve functioning, the use of findings and recommendations 
becomes a prime element in successful evaluation. 
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viewed in this manner, successful evaluation is heavily dependent 
upon the evaluator's sensitivity to the political context 
Influencing both the program development and evaluation. 
Although these forces are essentially beyond the direct control 
of evaluators, knowing about them can lead to actions In other 
areas, described below, that may have Impact on the political 
conditions for decision meUcers. 

Among the many factors that evaluators can control to a greater 
or lesser degree, and whose Impact on the utilization of findings 
and recommendations has been studied (Caplan, Morrison and 
Stambaugh, 1975; Collins, 1982; Levitan and Hughes, 1981), major 
emphasis has been directed at an examination of the 
characteristics of: 

1) the evaluation process; 

2) evaluator/user relationships; 

3) evaluation reports; 

4) evaluators; and 

5) users of evaluation information. 

From the research and the accumulated experience In the conduct 
of program evaluation by noted experts in the field, the factors 
most closely related to the use of evaluation Infor^atioffi appear 
to be 2) and 5) : characteristics of evaluater/user 
relationships, and characteristics of users of evaluation 
information. 

Surprisingly the quality of an evaluation design, its procedures 
and reporting, has not been shown to be significantly related to 
utilization (Patton et al, 1977). This apparently is due to the 
fact that users of evaluation information, and large, are not 
knowledgeable about research methodology and remain unaware of 
differences in quality along this dimension. This places great 
responsibility on the evaluator to ensure that the evaluation 
design is sound and capable of adequately providing 
answers to the questions that decision makers have. 

Another unexpected finding has been that whether the evaluation 
findings and recommendations are mainly negative or mainly 
positive about a program does not appear to affect utilization. 
What has been found more important for utilization is that 
recommendations do not come as a surprise — if they do there is 
a higher likelihood of rejection. This emphasizes the importance 
of communication between evaluators and users, suggested in 2) 
above, and especially preparing the potential user of information 
to all possible outcomes. 
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Productive relationships between evaluators and users of 
^Zfi?** ?" information are especially crucial for achieving 
utilization of findings and recomaendations . Fundamental to 
greater utilization is the involvement of program staff in the 
process at several points throughout its life-cycle, 
particularly involvement of the highest possible level of 
decision makers. Effective involvement of program staff (i.e. 
involvement which creates a higher level of acceptance and 
utilization of findings and recommendations) takes several forms, 
including: ' 

A. Forming evaluation objectives and questions to be 
addressed ; 

B. Identifying other people who should be involved in 
the development of questions and objectives; 

C. Identifying the most important sefcients of the study 
population from whom to derive information; 

D. Reviewing evaluation instruments to achieve correct 
content and tc serve the many audiences needing 
evaluation information; 

E. Top memagemeTit receiving rapid feedback from 
evaluators 2a>out problems encountered; 

P. Top manag^ent reviewing data interpretation prior to 
final report, preparing "minority report" to be 
included; 

G. All program levels receiving findings and 
recommendations; and 



H. Maintaining contact with evaluator through 
dissemination and implementation of findings and 
recommendations. 

All of these examples of a collaborative, adaptive and reciprocal 
relationship between the evaluator and the users of information 
2-® J^^^ suggested as having positive effects. Sowe of these 
effects are; 

A. Reduction of the element of surprise of negative 
fi^ndings; 

B. Development of trust in the evaluator on the part 
of the users; 

C. Increasing commitment of managers to utilization of 
evaluation information; 

D. Increasing the probability that recommended actions 
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appear to the aumagers to be clearly related to 
findings; and. 



Greater range of program staff receive comnunications 
about results, opening a wider variety of channels 
for encouraging utilization. 

Over all, it appears clear from the available literature on the 
subject that, without comaitaent and energy dedicated to 
evaluation on the part of program management and staff, 
utilization of evaluation findings jmd recommendations will be 
limited, at best. What is necessary are identifiable people who 
personally care about the evaluation and the information it 
generates. ifhere such people are present, evaluations are used; 
where absent, there is a correspondingly marked absence of 
evaluation impact. Users of evaluations are people who "are 
actively seeking information to reduce decision uncertainties so 
as to increase their ability to predict the outcomes of 
programmatic activity and to enhance their own discretion as 
decisionmakers; (Patton, 1978, p. 68). 

What also appears clear is that evaluators can take positive 
action to improve the probability of utilization without 
compromising- the objectivity of the evaluation. in fact, by 
involving the program staff and managers at all points in the 
process, the evaluations become more attuned to the reality of 
the program and are, in that way, more, rather than less, 
objective. ' 
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I - — 

The School Board of Dade County, Flf-ida adheres 
to a policy of nondiscrimination In educational 
programs/activities and employment and strives af- 
firmatively to provide equal opportunity for all 
as required by: 

Title VI of The Civil Rights Act of 1964 - pro- 
hibits discrimination on the basis of race, color, 
religion, or national origin. 

Title VII of The Civil Rights Act of 1964, as 
amended - prohibits discrimination In employment 
on the basis of race, color, religion, sex, or 
national origin. 

Title IX of the education amendments of 1972 - 
proh1!1^1ts d1scr1m1nat1or« on the basis of sex. 

Age OlSiOrlmlnatlon Act of 1967, as amended - pro- 
hibits discrimination on the basis of age between 
40 and 70. 

Section 504 of the Rehabilitation Act of 1973 - 
prohibits discrimination against the handicapped. 

Veterans are provided re-employment rights In <ic- 
cord nc<& with P.L. 93-508 (Federal) and Florida 
Stat^ Lawv Chapter 77-422, which also stipulates 
cateyorlcai preferences for employment. 
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